Programming interface operations in a driver in communication with a port for reinitialization of storage controller elements

ABSTRACT

A driver of a host bus adapter of a storage controller performs hardware resets of buses and other logic to which an embedded port of the host bus adapter is connected, in a first period of quiescing of I/O operations in the embedded port. The driver transmits one or more commands to the embedded port to resume selected I/O operations in the embedded port. A reinitialization of the driver is performed during a second period of quiescing of I/O operations in the embedded port, prior to sending a command to allow normal I/O operations in the embedded port.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/870,393 filed Sep. 30, 2015, which application is incorporated hereinby reference in its entirety.

BACKGROUND 1. Field

Embodiments relate to the programming of interface operations in adriver in communication with a port for reinitialization of storagecontroller elements.

2. Background

A storage controller may control access to storage for one or more hostcomputational devices that may be coupled to the storage controller overa network. A storage management application that executes in the storagecontroller may manage a plurality of storage devices, such as diskdrives, tape drives, flash drives, etc., that are coupled to the storagecontroller. A host may send Input/Output (abbreviated as I/O or IO)commands to the storage controller and the storage controller mayexecute the I/O commands to read data from the storage devices or writedata to the storage devices.

A host bus adapter (HBA) may comprise a circuit board and/or integratedcircuit based adapter that may include components such as a FibreChannel interface chip, where the Fibre Channel interface chip may bereferred to as an embedded port. The host bus adapter may provide I/Oprocessing and provide physical connectivity for the storage controllerto a storage area network (SAN), where the storage area network includesa Fibre Channel switched fabric. The storage controller (via the hostbus adapter) may act as a target that receives I/O commands from the oneor more host computational devices, where the one or more hostcomputational devices act as initiators of the I/O commands.

Communication between the hosts and the storage controller may occurover a Fibre Channel (FC) network, where Fibre Channel refers to anintegrated set of architectural standards for data transfer beingdeveloped by the American National Standards Institute. Fibre Channel isa high-speed network technology primarily used for storage areanetworks. Fibre Channel Protocol (FCP) is a transport protocol thatpredominantly supports transports commands over Fibre Channel networks.

Fibre Channel may be split into five layers: a Protocol-mapping layer(FC-4), a common service layer (FC-3), a network layer (FC-2), a datalink layer (FC-1), and a FC-0 layer that defines the physical link inthe system, including the fibre, connectors, optical and electricalparameters for a variety of data rates. Layers FC-0 through FC-2 arealso known as FC-PH, the physical layers of Fibre Channel, whereas FC-3and FC-4 layers define how Fibre Channel ports interact withapplications in computational devices. The FC-3 level of the FC standardis intended to provide the common services for features such asstriping, multicasting, etc.

FC-4, the highest layer in Fibre Channel, defines the applicationinterfaces that execute over Fibre Channel. FC-4 specifies the mappingrules of upper layer protocols using the FC layers below. FC-4 is formedby a series of profiles that define how to map legacy protocols to FibreChannel. Fibre Channel is capable of transporting both network andchannel information, and profiles for network and channel protocols,such as, Small Computer System Interface (SCSI), Intelligent PeripheralInterface (IPI), High Performance Parallel Interface (HIPPI) FramingProtocol. Internet Protocol (IP), Link Encapsulation (FC-LE),Single-Byte Command Code Set Mapping (SBCCS), etc., may be specified orproposed as protocol mappings in FC-4.

Fibre Connection (FICON) is a protocol of the fibre channel architectureand may also be referred to by the formal name of FC-SB-5. FICON is aprotocol layer that builds upon the Fibre Channel transport protocol.Further details of Fibre Channel protocol mapping for the Single-ByteCommand Code Sets may be found in the publication, “Fibre ChannelSingle-Byte Command Code Sets Mapping Protocol-5 (FC-SB-5)”, Rev. 2.0,published by the American National Standards Institute on Mar. 26, 2013.

The basic building blocks of a Fibre Channel connection are called“Frames”. The frames contain the information to be transmitted(Payload), the address of the source (i.e., initiator) and destination(i.e., target) ports and link control information. Frames are broadlycategorized as data frames and link control frames. Details of framingand signaling aspects of Fibre Channel may be found in the publication,“Fibre Channel Framing and Signaling-4 (FC-FS-4)”, Rev. 1.20, publishedby the American National Standard for Information Technology on Jul. 21,2015. Details of link services aspects of Fibre Channel may be found inthe publication, “Fibre Channel Link Services (FC-LS-3)”, Rev. 3.10,published by the American National Standard for Information Technologyon Feb. 1, 2014. The Fibre Channel Protocol for SCSI Fourth Version(FCP-4) standard describes the frame format and protocol definitionsrequired to transfer commands and data between a SCSI (Small ComputerSystem Interface) initiator and target using the Fibre Channel family ofstandards. Further details of FCP-4 may be found in the publication,“Information Technology-Fiber Channel Protocol for SCSI. Fourth Version(FCP-4), Revision 02b” published by the International Committee forInformation Technology Standards, on Jan. 3, 2011.

The storage controller may include a plurality of host bus adapters,where each host bus adapter may include a Fibre Channel Interface chipthat is an interface to switches that allow communication over a FibreChannel network between the storage controller and the plurality ofhosts.

Fibre Channel storage area networks may use the Fibre Channel protocol(used by the hardware to communicate), the SCSI protocol (used bysoftware applications to communicate to disks), and other protocols forcommunication. In Fibre channel, network connections are establishedbetween node ports (N_Ports) that are there in computers, servers,storage controllers, storage devices, printers, etc., and fabric ports(F_Ports) that are there in the Fibre channel switched fabric. A FibreChannel switched fabric relies on one or more switches to establishdirect, point-to-point connections between the source and targetdevices. Each Fibre Channel interface chip in the host bus adapters ofthe storage controller comprises a port that allows communication of thestorage controller to the hosts over the Fibre Channel switched fabric.

SUMMARY OF THE PREFERRED EMBODIMENTS

Provided are a method, a system, and a computer program product in whicha driver of a host bus adapter of a storage controller performs hardwareresets of buses and other logic to which an embedded port of the hostbus adapter is connected, in a first period of quiescing of I/Ooperations in the embedded port. The driver transmits one or morecommands to the embedded port to resume selected I/O operations in theembedded port. A reinitialization of the driver is performed during asecond period of quiescing of I/O operations in the embedded port, priorto sending a command to allow normal I/O operations in the embeddedport.

In further embodiments, in response to determining that an error hasoccurred in the storage controller, a first set of commands istransmitted to the embedded port of the host bus adapter to cause theembedded port to enter into the first period of quiescing of I/Ooperations, wherein the hardware resets of buses and other logic towhich the embedded port is connected are performed during the firstperiod. Configuration space registers of the buses and other logic arerestored to a state prior to performing of the hardware resets, prior totransmitting the one or more commands to the embedded port to resume theselected I/O operations.

In additional embodiments, subsequent to resumption and completion ofthe selected I/O operations, a second set of commands is transmitted tothe embedded port to cause the embedded port to enter into the secondperiod of quiescing of I/O operations, wherein the reinitialization ofthe driver is performed during the second period.

In certain embodiments, the transmitted first set of commands comprisesrequests to perform operations in the embedded port including: quiescingprocessing of received frames from a link; latching states of selectedhardware inputs; completing active direct memory access into drivermemory; deferring processing of link transitions; stopping all accessesto driver memory; and dequeing any messages to be sent to the driver.

In further embodiments, the one or more commands to resume selected I/Ooperations in the embedded port include requests to: resume processingof received frames from a link; enable detection and processing of linktransitions; requeue any held messages to the driver; and send anotification message to each response queue indicating completion of theselected I/O operations.

In additional embodiments, the second set of commands comprise optionsto allow partial direct memory access (DMA) activity to the drivermemory and to synchronize queue pointers, wherein the second set ofcommands requests the embedded port to: stop processing of receivedframes from a link; complete active DMA into driver memory; deferprocessing link transitions; stop a majority of accesses to drivermemory, while allowing access to memory extensions provided to theembedded port by the driver; and synchronize queue pointers on requestqueues by discarding any messages on the request queues and updating therequest queue out pointers.

In further embodiments, the second set of commands further comprisessending a message to the embedded port to terminate any remainingexchanges including operations to: send abort sequence (ABTS) for everyopen exchange without sending response messages to the driver; not sendABTS to host systems that have been indicated in port control block tonot receive ABTS; access host memory for offloaded exchanges todetermine state information; and relinquish control of all buffer andI/O control block (IOCB) resources associated with the driver.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 illustrates a block diagram of a computing environment comprisinga storage controller that includes one or more host adapters with one ormore Fibre Channel interface chips to couple the storage controller to aFibre Channel fabric to communicate with a plurality of hosts, inaccordance with certain embodiments:

FIG. 2 illustrates a first flowchart that show operations in a first anda second quiescence period of a driver and an embedded port using anapplication programming interface for reinitialization of storagecontroller elements, in accordance with certain embodiments;

FIG. 3 illustrates a second flowchart that shows operations of a driverand an embedded port that use the application programming interface forreinitialization of storage controller elements, in accordance withcertain embodiments;

FIG. 4 illustrates a third flowchart that operations of a driver and anembedded port that use the application programming interface forreinitialization of storage controller elements, in accordance withcertain embodiments;

FIG. 5 illustrates a fourth flowchart that shows operations of a driverand an embedded port that use the application programming interface forreinitialization of storage controller elements, in accordance withcertain embodiments:

FIG. 6 illustrates a fifth flowchart that shows operations of a driverand an embedded port that use the application programming interface forreinitialization of storage controller elements, in accordance withcertain embodiments;

FIG. 7 illustrates a sixth flowchart that shows operations of a driverand an embedded port that use the application programming interface forreinitialization of storage controller elements, in accordance withcertain embodiments:

FIG. 8 illustrates a seventh flowchart that shows operations of a driverthat uses the application programming interface for reinitialization ofstorage controller elements, in accordance with certain embodiments;

FIG. 9 illustrates a eighth flowchart that shows operations of anembedded port that uses the application programming interface forreinitialization of storage controller elements, in accordance withcertain embodiments;

FIG. 10 illustrates a block diagram of a cloud computing environment, inaccordance with certain embodiments;

FIG. 11 illustrates a block diagram of further details of the cloudcomputing environment of FIG. 10, in accordance with certainembodiments; and

FIG. 12 illustrates a block diagram of a computational system that showscertain elements that may be included in the storage controller, thehost bus adapter, the embedded port, and the host shown in FIG. 1, inaccordance with certain embodiments.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings which form a part hereof and which illustrate severalembodiments. It is understood that other embodiments may be utilized andstructural and operational changes may be made.

In enterprise storage control units, such as a storage controller,recovery from logic errors needs to be performed in a timely manner toavoid introducing noticeable increases in response times for recovery.In certain embodiments, a fast reset or warmstart process may be used toreinitialize the storage controller elements, such as servers and hostadapters, into a known state without performing a cold reset or initialprogram load. During this reset process the impact to active host I/O isminimized.

Certain embodiments minimize the impact to host I/O when performing afast reset on a host bus adapter, where the host bus adapter includes anembedded port that provides an interface to a Fibre Channel fabric,where the embedded port communicates with a driver of the host busadapter. The driver of the host bus adapter may communicate and controloperations of the embedded port. The communication mechanism between thedriver and the embedded port uses an application programming interface(API).

In certain embodiments, a host bus adapter directed multistep restartuses two quiesce periods for host I/O operations. In a first quiesceperiod, the embedded port stops processing any incoming frames and stopsdirect memory access (DMA) access from the embedded port. The firstquiesce period allows for the resetting of host adapter hardware such asa controller Application Specific Integrated Circuits (ASIC) and hotresets of Peripheral Component Interconnect Express (PCIe) buses towhich the embedded port is connected. A second quiesce process allowsfor reinitialization of the host adapter processor's internal structuresand state by reinitialization of the driver and subsequently normal I/Ooperations are resumed for the embedded port.

Exemplary Embodiments

FIG. 1 illustrates a block diagram of a computing environment 100comprising a storage controller 102 that includes one or more hostadapters 104 with one or more Fibre Channel interface chips 106 tocouple the storage controller 102 to a Fibre Channel fabric 108, tocommunicate with a plurality of hosts 110, 112, in accordance withcertain embodiments.

The storage controller 102 that includes the host bus adapter 104 maycontrol storage devices (not shown), and receive I/O commands from thehosts 110, 112. The storage controller 102 and the hosts 110, 112 maycomprise any suitable computational device including those presentlyknown in the art, such as, a personal computer, a workstation, a server,a mainframe, a hand held computer, a palm top computer, a telephonydevice, a network appliance, a blade computer, a processing device, etc.In certain embodiments the storage controller 102 may be comprised ofone or more storage servers. A plurality of storage servers may provideredundancy because if one storage server undergoes a failure from whichrecovery is not possible, an alternate storage server may perform thefunctions of the storage server that failed. The storage controller 102and the hosts 110, 112 may be elements in any suitable network, such as,a storage area network, a wide area network, the Internet, an intranet.In certain embodiments, the storage controller 102 and the hosts 110,112 may be elements in a cloud computing environment.

In FIG. 1, the storage controller 102 may include one or more host busadapters 104 that operate as targets of I/O operations initiated by oneor more hosts 110, 112. The host bus adapter 104 does not have controlover the arrival of host I/O operations. In certain embodiments, eachhost bus adapter 104 may be in the form of an adapter card that isplugged into the Peripheral Component Interconnect Express (PCIe) slotof the storage controller 102.

The host bus adapter 104 may include a PCIe bus 114 to which a host busadapter processor 116 and the Fibre Channel interface chip 106 arecoupled. An ASIC 118 may reside in the host bus adapter 104, where theASIC 118 provides a memory controller and PCIe bus connectivity.

The host bus processor 116 may be a single-core or a multi-coreprocessor. A driver 120 that supports upper level protocols e.g., FCP,FICON, FC-4 layer standards, etc., executes operations on the host busadapter processor 116. The driver 120 communicates with the FibreChannel interface chip 106 by using an application programming interface(API) 122. Various data structures, such as queues 124 are maintained bythe driver 120. In certain embodiments, the driver 120 may beimplemented in software, hardware, firmware or any combination thereof.

The Fibre Channel Interface Chip 106 is also referred to as an embeddedport. The embedded port 106 includes a processor 126 and a port firmware128 that supports lower level protocols like those for framing,signaling, etc. In certain embodiments, functions of the port firmware128 may be implemented in software, hardware, firmware or anycombination thereof. Various data structures, such as queues 130 (e.g.,request queues) are maintained by the port firmware 128. The embeddedport 106 supports lower level protocols of Fibre Channel and the driver120 supports upper level protocols. The embedded port 106 that supportslower level protocols of Fibre Channel connects the host bus adapter 104to the Fibre Channel fabric 108.

The hosts 110, 112 may send I/O commands to the storage controller 102over the Fibre Channel fabric 108. The embedded port 106 receives theFibre Channel frames corresponding to the request. The driver 120 whichsupports upper level protocols of Fibre Channel is in communication withthe embedded port 106. The driver 120 uses the embedded port 106 andcommunicates via the upper level protocols with the hosts 110, 112, andresponds to I/O commands via the embedded port 106.

Therefore, FIG. 1 illustrates certain embodiments in which a driver 120of a host bus adapter 104 in association with the port firmware 128 ofan embedded port 106 allows the host bus adapter 104 to process andrespond to I/O commands from one or more hosts 110, 112.

FIG. 2 illustrates a first flowchart 200 that show operations in a firstand a second quiescence period of a driver 120 and an embedded port 106using an application programming interface 122 for reinitialization ofelements of the storage controller 102, in accordance with certainembodiments.

In a first quiescence period 202, the driver 120 requests embedded port106 to quiesce (i.e., suspend) I/O operations, and during this firstperiod of quiescence the driver 120 resets (at block 204) host busadapter hardware, such as the ASIC 118 that provides memory controllerand PCIe bus connectivity, and performs (at block 206) hot resets ofinternal PCIe buses 114 to which the embedded port 106 is connected.Subsequently, the quiesced I/O operations are processed.

It should be noted that the operations of the first quiescence period202 resets the logic and hardware resources that may have been affectedby an error event, such that the logic and hardware resources are placedin a known state and may be used between the first quiescence period 202a second quiescence period 208 to process the incoming I/O requests, insuch a way that the number of incoming requests on a path from theinitiator to the storage target are reduced. For example, control unitbusy status or other messages such as process logout (PRLO) may be sentto initiators to slow the rate of incoming I/O requests (or stopincoming I/O requests) that may be discarded during the secondquiescence period 208. As a result, a reduction is made in I/Os duringreset events.

In a second quiescence period 108, the driver 120 is reinitialized andthe internal structures and the storage of the host bus adapter 104 arereinitialized. Subsequently, normal processing of I/O operations isperformed by the embedded port 106 via the port firmware 128.

FIG. 3 illustrates a second flowchart 300 that shows operations of adriver 120 and an embedded port 106 that use the application programminginterface for reinitialization of storage controller elements, inaccordance with certain embodiments. The driver operations 302 and theembedded port operations 304 are shown to the left and right of thedashed line 306 respectively.

Control starts at block 308 in which the driver 120, in response todetermining that an error has occurred in the storage controller 102that needs reinitialization of storage controller components, initiatesan entry into a first quiescence phase to perform hardware resets.Control proceeds to block 310 in which the driver 120 stops scanning ofInput/Output Control Blocks (IOCB) in message/response queues. Thedriver 120 sends (at block 311) a message to the embedded port 106 toflush current trace buffers from the port memory into driver memory forpreservation across hardware resets.

The embedded port 106 receives (at block 312) the message from thedriver 120 to flush trace buffers from the port memory into drivermemory. The embedded port 106 performs the operation to flush the tracebuffers from the port memory into the driver memory and responds (atblock 314) completion of the operation to the driver 120.

The driver 120 receives (at block 316) the response from the embeddedport 106 that the operation to flush trace buffers from the port memoryinto driver memory is complete. Control proceeds to block 318 in whichthe driver 120 sends a quiesce message to the embedded port 106 toperform operations associated with quiescing, including:

(1) the processing of received frames from the link;

(2) latching the state of hardware inputs needed for subsequent steps ofthe quiescing [e.g. latching the state of the Auto SCSI Status Inhibitand General Purpose IO (ASSI GPIO) input]:

(3) completing active direct memory access (DMA) into driver memory;

(4) deferring processing of link transitions;

(5) stopping all accesses to driver memory; and

(6) dequeuing any messages queued to be sent to the driver.

Control proceeds to block 320, which the embedded port 106 receives thequiesce message from the driver 106 to perform quiesce activities,including:

(1) processing of received frames from the link:

(2) latching the state of hardware inputs needed for subsequent steps ofthe quiescing;

(3) completing active direct memory access (DMA) into driver memory;

(4) deferring processing of link transitions:

(5) stopping all accesses to driver memory; and

(6) dequeuing any messages queued to be sent to the driver.

The embedded port 106 then responds (at block 322) to the driver 120that the activities have been quiesced. The driver 120 receives (atblock 324) the response from the embedded port 106 that activities havebeen quiesced and control proceeds to continuation block A 326 whichcontinues the flowchart in a subsequent figure.

FIG. 4 illustrates a third flowchart 400 that shows operations of adriver 120 and an embedded port 106 that use the application programminginterface for reinitialization of storage controller elements, inaccordance with certain embodiments. The driver operations 402 and theembedded port operations 404 are shown to the left and right of thedashed line 406 respectively.

From continuation block 326 control proceeds to block 408 in which thedriver 120 performs hardware resets including PCIe hot reset andrestores configuration space registers of the PCIe bus, etc., to thesame state as before the reset. The driver 120 sends (at block 410) amessage to the embedded port 120 to resume normal operations including:

(1) processing of received frames from the link;

(2) enabling of detection and processing of link transitions:

(3) requeuing of any held messages to the driver; and

(4) sending a notification message to each response queue indicatingcompletion of the resume process.

Control proceeds to block 412 in which the embedded port 106 receivesthe message from the driver 120 to resume normal operations including:

(1) processing of received frames from the link;

(2) enabling of detection and processing of link transitions:

(3) requeuing of any held messages to the driver; and

(4) sending a notification message to each response queue indicatingcompletion of the resume process.

The embedded port 106 responds (at block 416) for each response queue tothe driver 120 of the resume complete notification.

From block 410 control also proceeds to block 414 in which the driver120 sets a count of expected resume complete notification messages equalto the number of response queues currently configured. The driver 120sends (at block 418) a message to re-enable (i.e., enable once again)external trace capability. It may be noted that external trace goes intomemory outside of the embedded port 106. The embedded port 106 receives(at block 420) the message from the driver 120 to re-enable externaltrace capability, and in response re-enables external trace capabilityand responds (at block 422) that external trace capability has beenre-enabled. Control proceeds to block 424 in which the driver 120 entersa period of quiescing I/O activity.

From block 424 control proceeds to block 426 in which the driver 120resumes scanning of I/O control block (IOCB) message/response queues,and for each resume completion notification received, decrements countof expected resume complete messages till the count is decremented tozero. Control then proceeds to continuation block B 428 which continuesthe flowchart in a subsequent figure.

FIG. 5 illustrates a fourth flowchart 500 that shows operations of adriver 120 and an embedded port 106 that use the application programminginterface for reinitialization of storage controller elements, inaccordance with certain embodiments. The driver operations 502 and theembedded port operations 504 are shown to the left and right of thedashed line 506 respectively.

From continuation block 428 control proceeds to block 508 in which thedriver 120 initiates a second phase of quiescing embedded port activityto perform driver reinitialization. The driver 120 sends (at block 510)a message to the embedded port 106 to flush current trace buffers todriver memory for preservation across the driver initialization. Theembedded port 106 receives the message from the driver 120 to flushtrace buffers from port memory into driver memory, and the embedded port106 performs the flush operation before responding (at block 512).Control proceeds to block 514 in which the driver 120 sends a secondquiesce message to the embedded port 106 to quiesce activity, thismessage including options to allow partial direct memory access (DMA)activity to the driver memory and to synchronize queue pointers, wherethe second quiesce message requests the embedded port to:

(1) stop processing of received frames from the link:

(2) complete active DMA into driver memory:

(3) defer processing of link transitions;

(4) stop most accesses to driver memory, still allowing access to memoryextensions provided to the embedded port by the driver; and

(5) synchronize queue pointers on request queues by discarding anymessages on these queues and updating the request queue out pointers.

The embedded port 106 receives (at block 516) the second quiesce messagefrom the driver 120 to quiesce activity, this message including optionsto allow partial DMA activity to the driver memory and to synchronizequeue pointers, and performs:

(1) stopping processing of received frames from the link;

(2) completing active DMA into driver memory;

(3) deferring processing of link transitions:

(4) stopping most accesses to driver memory, still allowing access tomemory extensions provided to the embedded port by the driver; and

(5) synchronizing queue pointers on request queues by discarding anymessages on these queues and updating the request queue out pointers.

The embedded port 106 then responds (at block 518) to the second quiescemessage from the driver 120 and control proceeds to continuation block C520 which continues the flowchart in a subsequent figure.

FIG. 6 illustrates a fifth flowchart 600 that shows operations of adriver 120 and an embedded port 106 that use the application programminginterface for reinitialization of storage controller elements, inaccordance with certain embodiments. The driver operations 602 and theembedded port operations 604 are shown to the left and right of thedashed line 606 respectively.

Control proceeds from continuation block C 520 block 608 in which thedriver 120 sends a message to embedded port 106 to terminate anyremaining exchanges including operations to:

(1) send an abort sequence (ABTS) for every open exchange withoutsending response messages to the driver:

(2) not send ABTS to host systems that have been indicated in portcontrol block to not receive ABTS;

(3) access host memory for offloaded exchanges to determine their state;and

(4) relinquish control of all buffer and IOCB resources associated withthe driver.

Control proceeds to block 610 in which the embedded port 120 receives amessage from the driver to terminate any remaining exchanges, and inresponse performs:

(1) sending ABTS for every open exchange without sending responsemessages to the driver:

(2) not sending ABTS to host systems that have been indicated in portcontrol block to not receive ABTS;

(3) accessing host memory for offloaded exchanges to determine theirstate; and

(4) relinquishing control of all buffer and IOCB resources associatedwith the driver, and then the embedded port 106 responds to theterminate message from the driver 120 upon completion.

The driver 120 sends (at block 612) a message to the embedded port 120to resume normal operation including

(1) processing of received frames from the link;

(2) enabling of detection and processing of link transitions; and

(3) send a notification message to each response queue indicatingcompletion of the resume process.

The embedded port 106 receives (at block 614) a message from the driver120 to resume normal operation, and performs:

(1) processing of received frames from the link;

(2) enabling of detection and processing of link transitions; and

(3) sending a notification message to each response queue indicatingcompletion of the resume process.

The embedded port 106 also responds (at block 614) to the resume normaloperation message request from the driver 120. The driver 120 sets (atblock 616) a count of expected resume complete notification messagesequal to the number of response queues currently configured. Controlproceeds to continuation block D 618 which continues the flowchart in asubsequent figure.

FIG. 7 illustrates a sixth flowchart 700 that shows operations of adriver 120 and an embedded port 106 that use the application programminginterface for reinitialization of storage controller elements, inaccordance with certain embodiments. The driver operations 602 and theembedded port operations 604 are shown to the left and right of thedashed line 606 respectively.

From continuation block D 702 control proceeds to block 708 in which thedriver 120 sends a message to the embedded port 106 to re-enable (i.e.,enable once again) external trace capability, The embedded port 106receives (at block 710) a message from the driver 120 to re-enableexternal trace capability, re-enables the external trace capability, andresponds to the driver 120 when the operation is complete.

Control proceeds to block 712 in which the driver 120 resumes scanningof I/O control block (IOCB) message response queues, for each resumecompletion notification message received, and decrements the count ofexpected resume complete notification messages. The driver 120 continuesdiscarding messages on each queue until a resume complete notificationis received on all queues. The driver 106 then resumes (at block 714)normal operation and the embedded port 120 also resumes normal operationin response to a command from the driver 106.

Therefore FIGS. 3-7 illustrate operations performed by the driver 120and the embedded port 106 to perform the reinitialization of storagecontroller components in two quiescing periods to minimize disruption toI/O activity in the host bus adapter 104 that is in communication withthe hosts 110, 112.

FIG. 8 illustrates a seventh flowchart 800 that shows operations of adriver 120 that uses the application programming interface 122 forreinitialization of storage controller elements, in accordance withcertain embodiments.

Control starts at block 802, in which the driver 120 determines that anerror has occurred in the storage controller 102 that requires areinitialization (i.e., a initialization once again) of storagecontroller elements. The driver 120 transmits (at block 804) a first setof commands to the embedded port 106 of the host bus adapter 104 tocause the embedded port 106 to enter into the first period of quiescingof l/O operations, where the hardware resets of buses 114 and otherlogic (e.g., ASIC 118) to which the embedded port 106 is connected areperformed during the first period.

The driver 120 then restores (at block 806) the configuration spaceregisters of the buses 114 and other logic 118 to a state that theconfiguration space registers were in prior to the performing of thehardware resets. The driver 120 then transmits (at block 808) one ormore commands to the embedded port 106 to resume selected I/O operationsin the embedded port 106.

Control proceeds to block 810, in which the driver 120 is reinitializedduring a second period of quiescing of I/O operations in the embeddedport 106. The driver 120 then sends (at block 812) a command to allownormal I/O operations in the embedded port 106.

Therefore, FIG. 8 illustrates certain operations performed by the driver120 for reinitialization of storage controller elements with minimalimpact to I/O operations.

FIG. 9 illustrates a eighth flowchart 900 that shows operations of anembedded port 106 that uses the application programming interface 122for reinitialization of storage controller elements, in accordance withcertain embodiments.

Control starts at block 902 in which the embedded port 106 receives froma driver 120 of the host bus adapter 104, a first set of commands toquiesce I/O operations in the embedded port 106 for a first period,wherein hardware resets of buses 114 and other logic 118 to which theembedded port 106 is connected is performed in the first period ofquiescing of I/O operations.

The embedded port 106 receives (at block 904) one or more commands toresume selected I/O operations in the embedded port 106, where the oneor more commands to resume selected I/O operations in the embedded port106 are received subsequent to configuration space registers of thebuses 114 and other logic 118 being restored to a state prior to theperforming of the hardware resets.

Control proceeds to block 906 in which the embedded port 106 receives asecond set of commands to quiesce I/O operations for a second period.The embedded port receives a command to allow normal I/O operations,subsequent to the driver 120 being initialized during the second periodof quiescing of I/O operations.

Therefore, FIG. 9 illustrates certain operations performed by the port106 for reinitialization of storage controller elements with minimalimpact to I/O operations.

FIGS. 1-9 illustrate certain embodiments in which the impact to host I/Owhen performing a fast reset on a host bus adapter 104 is minimized. Twoquiesce periods are used for host I/O operations. In a first quiesceperiod, the embedded port 106 stops processing any incoming frames andstops direct memory access (DMA) access from the embedded port 106. Thefirst quiesce period allows for the resetting of host adapter hardwaresuch as a controller ASIC 118 and hot resets of PCIe buses 114 to whichthe embedded port 106 is connected. A second quiesce process allows forreinitialization of the host adapter processor's 116 internal structuresand state by reinitialization of the driver 120 and subsequently normalI/O operations are resumed for the embedded port 106.

Cloud Computing Environment

Cloud computing is a model for enabling convenient, on-demand networkaccess to a shared pool of configurable computing resources (e.g.,networks, servers, storage, applications, and services) that can berapidly provisioned and released with minimal management effort orservice provider interaction.

Referring now to FIG. 10, an illustrative cloud computing environment 50is depicted. As shown, cloud computing environment 50 comprises one ormore cloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community.Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 10 are intended to be illustrative only and that computing nodes10 and cloud computing environment 50 can communicate with any type ofcomputerized device over any type of network and/or network addressableconnection (e.g., using a web browser).

Referring now to FIG. 11, a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 10) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 11 are intended to be illustrative only and embodiments ofthe invention are not limited thereto.

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include mainframes, in oneexample IBM zSeries* systems; RISC (Reduced Instruction Set Computer)architecture based servers, in one example IBM pSeries* systems; IBMxSeries* systems; IBM BladeCenter* systems; storage devices; networksand networking components. Examples of software components includenetwork application server software, in one example IBM WebSphere*application server software; and database software, in one example IBMDB2* database software. * IBM, zSeries, pSeries, xSeries, BladeCenter,WebSphere, and DB2 are trademarks of International Business MachinesCorporation registered in many jurisdictions worldwide.

Virtualization layer 62 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers;virtual storage; virtual networks, including virtual private networks;virtual applications and operating systems; and virtual clients.

In one example, management layer 64 may provide the functions describedbelow. Resource provisioning provides dynamic procurement of computingresources and other resources that are utilized to perform tasks withinthe cloud computing environment. Metering and Pricing provide costtracking as resources are utilized within the cloud computingenvironment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal provides access to the cloud computing environment forconsumers and system administrators. Service level management providescloud computing resource allocation and management such that requiredservice levels are met. Service Level Agreement (SLA) planning andfulfillment provide pre-arrangement for, and procurement of, cloudcomputing resources for which a future requirement is anticipated inaccordance with an SLA.

Workloads layer 66 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation; software development and lifecycle management; virtualclassroom education delivery; data analytics processing; transactionprocessing; and the reinitialization of storage controller elements 68as shown in FIGS. 1-9.

Additional Embodiment Details

The described operations may be implemented as a method, apparatus orcomputer program product using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof. Accordingly, aspects of the embodiments may take the form of anentirely hardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the embodiments may take the form of a computer programproduct. The computer program product may include a computer readablestorage medium (or media) having computer readable program instructionsthereon for causing a processor to carry out aspects of the presentembodiments.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present embodiments may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present embodiments.

Aspects of the present embodiments are described herein with referenceto flowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instruction.

FIG. 12 illustrates a block diagram that shows certain elements that maybe included in the host bus adapter 104, the storage controller 102, theembedded port 106, or other computational devices in accordance withcertain embodiments. The system 1200 may include a circuitry 1202 thatmay in certain embodiments include at least a processor 1204. The system1200 may also include a memory 1206 (e.g., a volatile memory device),and storage 1208. The storage 1208 may include a non-volatile memorydevice (e.g., EEPROM, ROM, PROM, flash, firmware, programmable logic,etc.), magnetic disk drive, optical disk drive, tape drive, etc. Thestorage 1208 may comprise an internal storage device, an attachedstorage device and/or a network accessible storage device. The system1200 may include a program logic 1210 including code 1212 that may beloaded into the memory 1206 and executed by the processor 1204 orcircuitry 1202. In certain embodiments, the program logic 1210 includingcode 1212 may be stored in the storage 1208. In certain otherembodiments, the program logic 1210 may be implemented in the circuitry1202. Therefore, while FIG. 12 shows the program logic 1210 separatelyfrom the other elements, the program logic 1210 may be implemented inthe memory 1206 and/or the circuitry 1202.

Certain embodiments may be directed to a method for deploying computinginstruction by a person or automated processing integratingcomputer-readable code into a computing system, wherein the code incombination with the computing system is enabled to perform theoperations of the described embodiments.

The terms “an embodiment”, “embodiment”. “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the present invention(s)” unless expressly specifiedotherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary a variety of optional components are described toillustrate the wide variety of possible embodiments of the presentinvention.

Further, although process steps, method steps, algorithms or the likemay be described in a sequential order, such processes, methods andalgorithms may be configured to work in alternate orders. In otherwords, any sequence or order of steps that may be described does notnecessarily indicate a requirement that the steps be performed in thatorder. The steps of processes described herein may be performed in anyorder practical. Further, some steps may be performed simultaneously.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the present inventionneed not include the device itself.

At least certain operations that may have been illustrated in thefigures show certain events occurring in a certain order. In alternativeembodiments, certain operations may be performed in a different order,modified or removed. Moreover, steps may be added to the above describedlogic and still conform to the described embodiments. Further,operations described herein may occur sequentially or certain operationsmay be processed in parallel. Yet further, operations may be performedby a single processing unit or by distributed processing units.

The foregoing description of various embodiments of the invention hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed. Many modifications and variations are possible in lightof the above teaching. It is intended that the scope of the invention belimited not by this detailed description, but rather by the claimsappended hereto. The above specification, examples and data provide acomplete description of the manufacture and use of the composition ofthe invention. Since many embodiments of the invention can be madewithout departing from the spirit and scope of the invention, theinvention resides in the claims hereinafter appended, affiliates.

What is claimed is:
 1. A method performed by a driver of a host busadapter of a storage controller, the method comprising: in response todetermining that an error has occurred in the storage controller,transmitting a first set of commands to an embedded port of the host busadapter to cause the embedded port to enter into a first period ofquiescing of I/O operations, wherein hardware resets of buses and otherlogic to which the embedded port is connected are performed during thefirst period; restoring configuration space registers of the buses andother logic to a state prior to performing of the hardware resets andsubsequently transmitting, by the driver, one or more commands to theembedded port to resume selected I/O operations in the embedded port,wherein the one or more commands to resume selected I/O operations inthe embedded port include requests to: resume processing of receivedframes from a link; enable detection and processing of link transitions;requeue any held messages to the driver; and send a notification messageto each response queue indicating completion of the selected I/Ooperations; subsequent to resumption and completion of the selected I/Ooperations, transmitting a second set of commands to the embedded portto cause the embedded port to enter into a second period of quiescing ofI/O operations; and performing reinitialization of the driver during thesecond period of quiescing of I/O operations in the embedded port. 2.The method of claim 1, wherein the transmitted first set of commandscomprises requests to perform operations in the embedded port including:quiescing processing of received frames from a link; latching states ofselected hardware inputs; completing active direct memory access intodriver memory; deferring processing of link transitions; stopping allaccesses to driver memory; and dequeing any messages to be sent to thedriver.
 3. The method of claim 1, wherein the second set of commandscomprise options to allow partial direct memory access (DMA) activity todriver memory and to synchronize queue pointers, and wherein the secondset of commands requests the embedded port to: stop processing ofreceived frames from a link; complete active DMA into driver memory;defer processing link transitions; stop a majority of accesses to drivermemory, while allowing access to memory extensions provided to theembedded port by the driver; and synchronize queue pointers on requestqueues by discarding any messages on the request queues and updating therequest queue out pointers.
 4. The method of claim 1, wherein the secondset of commands further comprises sending a message to the embedded portto terminate any remaining exchanges including operations to: send anabort sequence (ABTS) for every open exchange without sending responsemessages to the driver; not send any ABTS to host systems that have beenindicated in port control block to not receive any ABTS; access hostmemory for offloaded exchanges to determine state information; andrelinquish control of all buffer and I/O control block (IOCB) resourcesassociated with the driver.
 5. A system, comprising: a memory; and aprocessor coupled to the memory, wherein the processor performsoperations, the operations comprising: in response to determining thatan error has occurred, transmitting, by a driver, a first set ofcommands to an embedded port of a host bus adapter to cause the embeddedport to enter into a first period of quiescing of I/O operations,wherein hardware resets of buses and other logic to which the embeddedport is connected are performed during the first period; restoringconfiguration space registers of the buses and other logic to a stateprior to performing of the hardware resets and subsequentlytransmitting, by the driver, one or more commands to the embedded portto resume selected I/O operations in the embedded port, wherein the oneor more commands to resume selected I/O operations in the embedded portinclude requests to: resume processing of received frames from a link;enable detection and processing of link transitions; requeue any heldmessages to the driver; and send a notification message to each responsequeue indicating completion of the selected I/O operations; andsubsequent to resumption and completion of the selected I/O operations,transmitting a second set of commands to the embedded port to cause theembedded port to enter into a second period of quiescing of I/Ooperations; and performing reinitialization of the driver during thesecond period of quiescing of I/O operations in the embedded port. 6.The system of claim 5, wherein the transmitted first set of commandscomprises requests to perform operations in the embedded port including:quiescing processing of received frames from a link; latching states ofselected hardware inputs; completing active direct memory access intodriver memory; deferring processing of link transitions; stopping allaccesses to driver memory; and dequeing any messages to be sent to thedriver.
 7. The system of claim 5, wherein the second set of commandscomprise options to allow partial direct memory access (DMA) activity todriver memory and to synchronize queue pointers, wherein the second setof commands requests the embedded port to: stop processing of receivedframes from a link; complete active DMA into driver memory; deferprocessing link transitions; stop a majority of accesses to drivermemory, while allowing access to memory extensions provided to theembedded port by the driver; and synchronize queue pointers on requestqueues by discarding any messages on the request queues and updating therequest queue out pointers.
 8. The system of claim 5, wherein the secondset of commands further comprises sending a message to the embedded portto terminate any remaining exchanges including operations to: send anabort sequence (ABTS) for every open exchange without sending responsemessages to the driver; not send any ABTS to host systems that have beenindicated in port control block to not receive any ABTS; access hostmemory for offloaded exchanges to determine state information; andrelinquish control of all buffer and I/O control block (IOCB) resourcesassociated with the driver.
 9. A computer program product, the computerprogram product comprising a computer readable storage medium havingcomputer readable program code embodied therewith, the computer readableprogram code configured to perform operations, the operationscomprising: in response to determining that an error has occurred in sstorage controller, transmitting, by a driver, a first set of commandsto an embedded port of a host bus adapter to cause the embedded port toenter into a first period of quiescing of I/O operations, whereinhardware resets of buses and other logic to which the embedded port isconnected are performed during the first period; restoring configurationspace registers of the buses and other logic to a state prior toperforming of the hardware resets and subsequently transmitting, by thedriver, one or more commands to the embedded port to resume selected I/Ooperations in the embedded port, wherein the one or more commands toresume selected I/O operations in the embedded port include requests to:resume processing of received frames from a link; enable detection andprocessing of link transitions; requeue any held messages to the driver;and send a notification message to each response queue indicatingcompletion of the selected I/O operations; and subsequent to resumptionand completion of the selected I/O operations, transmitting a second setof commands to the embedded port to cause the embedded port to enterinto a second period of quiescing of I/O operations; and performingreinitialization of the driver during the second period of quiescing ofI/O operations in the embedded port.
 10. The computer program product ofclaim 9, wherein the transmitted first set of commands comprisesrequests to perform operations in the embedded port including: quiescingprocessing of received frames from a link; latching states of selectedhardware inputs; completing active direct memory access into drivermemory; deferring processing of link transitions; stopping all accessesto driver memory; and dequeing any messages to be sent to the driver.11. The computer program product of claim 9, wherein the second set ofcommands comprise options to allow partial direct memory access (DMA)activity to driver memory and to synchronize queue pointers, wherein thesecond set of commands requests the embedded port to: stop processing ofreceived frames from a link; complete active DMA into driver memory;defer processing link transitions; stop a majority of accesses to drivermemory, while allowing access to memory extensions provided to theembedded port by the driver; and synchronize queue pointers on requestqueues by discarding any messages on the request queues and updating therequest queue out pointers.
 12. The computer program product of claim 9,wherein the second set of commands further comprises sending a messageto the embedded port to terminate any remaining exchanges includingoperations to: send an abort sequence (ABTS) for every open exchangewithout sending response messages to the driver; not send any ABTS tohost systems that have been indicated in port control block to notreceive any ABTS; access host memory for offloaded exchanges todetermine state information; and relinquish control of all buffer andI/O control block (IOCB) resources associated with the driver.